
Assessing the Relationship 
Between Questioning and 
Understanding to Improve 
Learning and Thinking (QUILT) 
and Student Achievement in 
Mathematics: A Pilot Study 



Jim Craig 
Leslie Cairo, III 

December 2005 



Appalachia Educational Laboratory (AEL) 
at 



EDV^MTI/^v 



Partners in education. Focused on resuits. 



Assessing the Relationship Between Questioning and 
Understanding to Improve Learning and Thinking (QUILT) and Student 
Achievement in Mathematics: A Pilot Study 



By 

Jim Craig 
Leslie Cairo, III 



Appalachia Educational Laboratory at Edvantia, Inc. 
Charleston, West Virginia 



December 2005 




Edvantia was founded in 1966 as the Appalachia Educational Laboratory, Inc. (AEL); on 
September 1, 2005, AEL became Edvantia, Inc. The Regional Educational Laboratory for the 
Appalachian region is known as the Appalachia Educational Laboratory at Edvantia. 



Edvantia is a nonprofit education research and development corporation, founded in 1966, that 
partners with practitioners, education agencies, publishers, and service providers to improve 
learning and advance student success. Edvantia provides clients with a range of services, 
including research, evaluation, professional development, and consulting. 



Eor information about Edvantia research, products, or services, contact 



P.O. Box 1348, Charleston, WV 25325 • 304.347.0400 • 800.624.9120 • fax 304.347.0487 
One Vantage Way, Suite D-210, Nashville, TN 37228 • 615.565.0101 • fax 615.565.0112 

info@edvantia.org • www.edvantia.org 



© 2005 by Edvantia, Inc. 

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, 
or transmitted in any form or by any means — electronic, mechanical, photocopying, recording, or 
otherwise — without the prior permission of Edvantia. 



Edvantia maintains policies and procedures that govern the Institutional Review Board in 
ensuring the welfare and protecting the rights of human subjects who choose to participate in 
research and evaluation activities. 



This publication is based on work sponsored wholly or in part by the Institute of Education 
Sciences (lES), U.S. Department of Education, under contract number ED-Ol-CO-0016 Its 
contents do not necessarily reflect the positions or policies of lES, the Department, or any other 
agency of the U.S. government. 



Edvantia is an equal employment opportunity/affirmative action employer. 




Contents 



Abstract vi 

Introduction 1 

Methodology 3 

Participants 3 

QUILT Implementation 3 

QUILT Coding System 4 

Coding Procedure 5 

Mathematics Achievement Assessment 5 

Data Analysis 6 

Results 7 

Conclusions 16 

QUILT Behaviors 16 

Gains in Mathematics Achievement 16 

Next Steps 17 

References 18 

Appendix: Description of QUILT Program 



iii 




List of Tables 



Table 1 QUILT Coding System Summary 4 

Table 2 Percentage of Questions Asked by the Teacher and Student During the 

Instructional Lesson in Mathematics 7 



IV 




List of Figures 



Figure 1 Percentage of questions of different lengths of Wait Time I asked in an 

average instructional session 8 

Figure 2 Percentage of questions answered by different individuals or groups in the 
classroom 9 

Figure 3 Percentage of different types of questions asked by teachers 10 

Figure 4 Percentage of answers given by the teacher, the whole class, the 

designated student, an unknown student, or “no response” 11 

Figure 5 Percentage of answers of different types given in responses to teacher 

questions 12 

Figure 6 Percentage of responses with no Wait Time II to more than 3 seconds 13 

Figure 7 Average number of different types of teacher responses to students’ 

answers to teacher questions 14 

Figure 8 Percentage of different types of student responses to teacher 

questions 15 



V 




Abstract 



Good questions, effectively delivered are a key feature of learning and teaching. QUILT is a 
research-based, field-tested professional development program designed to assist teachers in 
developing and implementing effective questioning skills to facilitate student learning. The 
QUILT framework presents classroom questioning as a five-stage process: (1) question 
preparation, (2) presentation of questions, (3) prompting, (4) processing of student responses, 
and (5) reflection on questioning practice. A pilot study of the effects of QUILT on student 
achievement was conducted to explore the feasibility of an efficacy study of QUILT and to 
identify the methodological issues that would have to be addressed in its conduct. Twenty-eight 
fifth- and sixth-grade elementary school teachers in a rural school district in Kentucky were 
trained to implement QUILT as a teaching/leaming strategy. During the spring of 2005, the 
teachers videotaped three different mathematics instructional sessions in which they used QUILT 
questioning techniques. Each tape was viewed by researchers from the Appalachia Educational 
Laboratory at Edvantia, Inc., and summarized using a special coding sheet. The QUILT 
behaviors were examined in relation to gains in mathematics achievement. This was 
accomplished using the mathematics section of the Measures of Academic Progress (Northwest 
Evaluation Association, 2005). The findings indicate that teachers implemented some QUILT 
questioning behaviors but not others in some of their mathematics instructional lessons and that a 
well controlled, randomized control trial is needed to examine the efficacy of QUILT as an 
effective instructional technique, being sure to address teacher training, fidelity of 
implementation, and assessment of student achievement issues. 




Introduction 



Good questions, effectively delivered, facilitate student learning and thinking as they 
serve to motivate and focus student attention, provide opportunities for practice and rehearsal, 
and provide the opportunity for teachers to assess how well students are mastering content 
(Dillon, 1988). Questioning is a core function of both learning and teaching; questions can 
stimulate students to think at higher cognitive levels (Dillon). 

The QUILT professional development program, developed by staff in 1990 from the 
Appalachia Educational Laboratory (AEL) at Edvantia, Inc., is research based in terms of both 

(1) teacher questioning behavior and (2) the design of the staff development program. Eirst, 
during development of program materials, the relevant empirical research literature (including 
relevant correlational, quasi-experimental, and experimental studies on effective questioning 
strategies) were thoroughly reviewed. Program materials have been periodically updated to 
assure they are consistent with the most rigorous empirical research available on questioning. 
Second, the research literature regarding professional development practices was reviewed and 
used in the development of the program. This research literature has periodically been updated to 
include the most current research regarding professional development. As a result, the QUILT 
program design focuses on student learning, is built on principles of adult learning, and is 
designed to be implemented over time. A general description of the QUILT program is contained 
in the appendix. 

The QUILT program has been field-tested (AEL, 1994). In the field test, 42 
participating schools were randomly assigned to one of three different treatment conditions: (1) 
Condition A — staff development training over an entire school year through implementation of 
all four components of the QUILT model: induction training, collegiums, partnering, and 
classroom application; (2) Condition B — three days of staff development training (i.e., the 
induction training) on the QUILT model; or (3) Condition C — three hours of staff development 
training on topics contained in the QUILT model. The researchers were able to determine that 
participating in the full QUILT program (Condition A) did positively affect the questioning skills 
of teachers. Specifically, the researchers found evidence that the full QUILT program design best 
helped teachers develop and use the QUILT questioning skills in their classrooms. 

The QUILT program presents the questioning-answering-reacting interaction between 
students and teachers as dynamic and interrelated systems whose parts interact to affect the 
outcome of the teaching-learning process (AEL, 1994). The core content of QUILT includes 
effective classroom questioning strategies that are research based and consistent with active 
learning and teaching for understanding (Perkins, 1998; Perrone, 1998). These are embedded in a 
framework that presents classroom questioning as a five-stage process: (1) question preparation, 

(2) presentation of questions, (3) prompting, (4) processing of student responses, and (5) 
reflection on questioning practice. Specific examples of teacher behaviors included in this 
framework include the following: 

(a) Wait Time I. Teachers typically require students to respond almost 

instantaneously to questions, allowing less than 1 second for students to think 
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through their answers. However, in classrooms where teachers wait 3 to 5 
seconds, students give longer responses, answer more frequently at higher levels 
of evaluation and synthesis, and ask more questions (McGlathery, 1978; Rowe, 
1986). 

(b) Wait Time II. Teachers usually react immediately to a student response, waiting 
an immeasurably short amount of time before providing feedback or making 
another instructional move. In classrooms where teachers wait 3 to 5 seconds 
after the initial student response, students answer more completely and more 
correctly; exhibit more speculative and inferential thinking; ask more questions; 
increase interactions with other students; and demonstrate more confidence in 
their responses (Garigliano, 1972; Gooding, Swift, & Swift, 1983; Rowe, 1974). 

(c) Asking questions at all cognitive levels. About 75% to 80% of the questions 
posed in both elementary and secondary classrooms are recall-of-knowledge 
questions (Dillon, 1988; Gall, 1984; McGlathery, 1978). However, when 
students are afforded opportunities to answer questions at higher cognitive 
levels, they demonstrate an ability to analyze, synthesize, and evaluate; they also 
score better on tests measuring recall and understanding of content (Redfield & 
Rousseau, 1981). 

(d) Redirecting questions. Teachers typically answer questions when students do 
not answer or do not give the answer the teacher was seeking. However, when 
teachers pose one question to more than one student (i.e., when they redirect 
questions), students are held more accountable for answering all questions; 
additionally, the interaction among and between students increases (Ornstein, 
1988; Riley, 1981). 

(e) Designating a respondent. Teachers frequently name a student to answer a 
question before posing the question. However, when teachers name a student 
after posing a question, all students are more likely to attend to the question and 
to prepare a covert response in preparation to being called upon to answer the 
questions (Gall, 1984; Ornstein, 1988). 

(f) Repeating student answers. Teachers commonly repeat student answers; 
however, when teachers do not repeat answers, students pay greater attention to 
and show increased respect for their classmates’ responses. 

It has been documented for teachers in 13 schools (AEL, 1994) that, after one year’s 
participation in QUILT, teachers showed significant gains in knowledge, understanding, and 
application of effective questioning, including (1) a decrease in the number of teacher questions, 
(2) an increase in wait time, (3) an increase in the number of redirected questions, (4) an increase 
in the percentage of questions directed at higher cognitive levels, (5) a decrease in the number of 
respondents designated before questions are asked, and (6) a decrease in repetition of student 
answers. 
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The next step in researching the effectiveness of QUILT is to examine the relationship 
between such changes in teacher behavior and student achievement. Therefore, a pilot study was 
conducted to explore the feasibility of such a study and to identify the methodological issues that 
would have to be addressed in a full-scale efficacy study of the effects of QUILT on student 
achievement. 



Methodology 

A pilot study was conducted in cooperation with a rural school district in Kentucky which 
implemented QUILT district-wide as an instructional strategy. Therefore, the study was 
descriptive in nature and based on assessing the correlation between the imposition of QUILT 
questioning behaviors and student achievement. 



Participants 

The 28 fifth- and sixth-grade elementary school teachers who participated in the study 
were from a rural school district in Kentucky. These teachers were trained to implement QUILT 
as a teaching/learning strategy. Some of the participants were among those who took part in the 
4-day comprehensive QUILT training conducted by ALL professional development staff; others 
were trained by school district personnel who had attended a training-of-trainers event conducted 
by ALL staff. No systematic evaluation assessments were conducted in association with the 
QUILT training conducted by school district personnel so there is no available evidence 
regarding the fidelity of QUILT training the teachers received. 



QUILT Implementation 

At the start of the 2003 school year, almost all teachers and administrators in the school 
district received three days of QUILT training through professional development workshops held 
at their schools. The trainers for these workshops were teachers and administrators who had been 
trained by ALL staff to train other teachers and administrators to use QUILT. Beginning in 
September 2003, the QUILT questioning technique was adopted by the school district as an 
instructional strategy to be used in classrooms across the district. The central office personnel 
who received the initial QUILT training from ALL staff were responsible for training teachers 
and administrators new to the district in the use of QUILT questioning behaviors in their schools. 

During the spring of 2005, fifth- and sixth- grade elementary school teachers from each 
of the district’s six elementary schools videotaped three different mathematics instructional 
sessions in which they used QUILT questioning techniques. The tapes averaged 14.10 minutes 
(SD=2.2) in length. Each tape was viewed by AEL researchers, who summarized each session by 
using a special coding sheet to record QUILT behaviors observed during the questioning session. 
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QUILT Coding System 



The teachers’ QUILT behaviors were categorized and summarized using a machine- 
readable coding system (summarized in Table 1 below) that captures who asks the question, who 
is asked the question, what kind of question is asked, who responds to the question, what type of 
response is given, what the teacher’s response is (if any), what the student reaction is (if any). 
Wait Time I, and Wait Time II. 



Table 1. QUILT Coding System Summary 



Coding Category 


Behaviors Coded 


Who asks the 
question 


• Teacher 

• Student 


To whom is the 
question addressed 


• Teacher 

• Designated student to whom the question is addressed before 
asking the question 

• Asks the question, then designates a respondent 

• Whole class 

• Addresses the question to one or more cooperative groups 


What kind of 
question is asked 


• Recall (facts from memory) 

• Check for understanding of procedures and routines 

• Use (using knowledge to comprehend, apply, or analyze) 

• Teacher repeats the question two or more times 

• Create (synthesizing to arrive at a conclusion) 

• Teacher asks multiple questions 


Who responds to the 
question 


• Teacher 

• Designated student responds 

• Whole class, choral response 

• None 

• Unknown 


What type of 
response is given 


• Recall 

• Use 

• Create 

• Asks for clarification 

• No answer 

• Unknown 
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Coding Category 


Behaviors Coded 


What is the teacher’s 
response 


• Positive feedback and/or praise (e.g.. That’s right.) 

• Negative feedback (e.g.. That’s wrong.) 

• Corrective feedback 

• Criticism 

• Teacher repeats the answer 

• Teacher repeats the question 

• Teacher probes 

• Teacher cues/prompts when answer is incomplete or incorrect 

• Teacher rephrases the answer 

• Teacher redirects the question 

• Teacher uses student response 

• None 


What is the student’s 
reaction 


• One student adds on to the answer given 

• More than one student adds onto the answer given 

• One student asks a question 

• More than one student asks a question 


Wait Time I 


• The estimated length of time in seconds between the teacher 
asking a question and acting further (e.g., calling upon a 
student) 


Wait Time II 


• The estimated length of time in seconds between a student’s 
response and the teacher’s reaction (e.g., providing positive 
feedback) 



Coding Procedure 



Each teacher’s videotaped session was coded by two AEL researchers using the machine- 
readable coding system to record QUILT behaviors observed during the questioning session. 

Any discrepancy in coding between the two researchers was resolved by both researchers 
reexamining the section in which the coding difference occurred and coming to agreement about 
how the particular behavior should be coded. There were no instances in which the raters could 
not come to an agreement. Therefore, interrater reliability was 100% for each coded behavior. 
Each videotaped questioning session required approximately one and one half hours to code to 
agreement. 



Mathematics Achievement Assessment 

The assessment of mathematics achievement was accomplished using the mathematics 
section of the Measures of Academic Progress (Northwest Evaluation Association, 2005), an 
achievement test already being administered by the cooperating district to assess student 
achievement in mathematics. This test has been created to measure what a student knows and 
needs to learn. The test is based on Rasch Units, referred to as RITs, which create an equal 
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interval scale that aligns student achievement levels with item difficulties. The higher the RIT 
score, the higher the level of difficulty. The equal interval scale is grade independent, and the 
scores mean the same thing as when the scale was first introduced. Most students’ scores fall 
between the RIT score values of 140 and 300. 

Test-retest reliabilities for the Measures of Academic Progress mathematics test have 
been found to range from .86 to .93 (Northwest Evaluation Association, 2004). For Grades 4, 5, 
and 6 on the same test, marginal reliability coefficients, which are indexes of reliability based on 
combining measurement error estimated at different points on a scale, have been reported to be 
.94. This method of estimating reliability is purported to yield estimates nearly identical to 
coefficient alpha (Northwest Evaluation Association, 2004). 

The concurrent validity of the Measures of Academic Progress mathematics test has been 
examined by comparing Measures of Academic Progress mathematics RIT scores with 
mathematics scores on various states’ achievement tests (e.g., the Arizona Instrument to Measure 
Standards and the Illinois Standards Achievement Test) and the Stanford Achievement Test, 9^’’ 
edition. For Grades 4, 5, and 6, correlations of .80 to .89 have been found between the Measures 
of Academic Progress mathematics test and these other measures of mathematics achievement 
(e.g., the Stanford Achievement Test, 9* edition). The validity of the Measures of Academic 
Progress mathematics test as a proxy measure for the Kentucky Core Content Test of student 
achievement in mathematics is not known. 



Data Analysis 

The completed QUILT coding sheets were scanned, and SPSS data files were created for 
analysis. Two different types of analyses were then conducted. First, descriptive statistics for the 
various coding categories were produced and graphic displays of these outcomes created. 
Second, correlation and regression analyses were conducted to assess degrees of association 
between coded QUILT behaviors and the average mathematics achievement of the class as 
assessed by Measures of Academic Progress (Northwest Evaluation Association, 2005). 
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Results 



The reporting of the results has been organized such that for each research question 
addressed, descriptive statistics are reported and graphic presentations are provided, if 
appropriate, to summarize teacher questioning behavior. Second, the results of statistical 
analyses are reported to describe the degree of association between specific teacher questioning 
behaviors and mathematics achievement. 

Research Question 1: Who asks most questions during a QUILT session? Is there a 
relationship between who asks the questions and average gains in mathematics 
achievement test scores? 

Virtually all questions (99.8%) in the videotaped sessions were asked by teachers. 
Students were rarely, if ever, observed asking questions of the teacher or other students. This 
information is summarized in Table 2 below. 

Table 2. Percentage of Questions Asked by the Teacher and Student During the Instructional 
Lesson in Mathematics 



Individual Who Asked 
the Question 


Percentage of Questions 
Asked 


Student 


0.02% 


Teacher 


99.8% 



With virtually no variance, no inferential statistical analyses were conducted. 

Research Question 2: After a question is asked, is there use of Wait Time I? Is there 
a relationship between the use of Wait Time I and average gains in mathematics 
achievement test scores? 

Wait Time I captures the amount of time teachers allow students to think before asking 
them to respond. On average, teachers asked approximately 42 questions in a 15-minute period. 
As is shown in Figure 1, about 50% of the time, on average, the teachers gave the students no 
Wait Time I (labeled None on the graph). About a third of the time, on average, the teachers 
gave the students more than 3 seconds of Wait Time I, an ideal wait time in the QUILT model. 
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3 sec 

Wait Time I (in seconds) 



Figure 1. Percentage of questions of different lengths of Wait Time I asked in an average 
instructional session 



The correlation of a teacher’s average Wait Time I across three instructional sessions and 
mean gain in mathematics achievement was small and nonsignificant, r = .075, p = .704. This 
indicates that for the sample studied, there is no association between the average amount of Wait 
Time I teachers provided and average gains in mathematics achievement as assessed by the 
Measures of Academic Progress. 

Research Question 3: Who is typically designated to answer most questions during a 
QUILT session? Is there an association between the designee and average gains in 
mathematics achievement test scores? 

The designation of a student to answer a question after a question is posed is an 
indication that a teacher is engaging in a QUILT behavior. About 60% of the time, on average, 
students were designated to answer a question after the teacher posed a question. Almost 20% of 
the time, on average, the whole class was called upon by the teacher to answer a question in a 
choral response. About 20% of the time, students were identified by the teacher as the individual 
who was to answer a question before the question was asked by the teacher. This information is 
presented in Figure 2. 
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Figure 2. Percentage of questions answered by different designated individuals or groups 
in the classroom 



The correlation between the average percentage of questions per session for a teacher 
where students were designated to answer after the question was posed and average gain in 
mathematics achievement for each class was found to be low and nonsignificant, r = -.056, p = 
.776. The elimination of an outlier from the analysis increased the correlation, but it remained 
nonsignificant, r = .188, p = .348. Thus, there is no evidence that engaging in the QUILT 
behavior designating students to answer after a question was posed is related to average gains in 
mathematics achievement as assess by the Measures of Academic Progress. 

Research Question 4: What level of question is asked most frequently during a 
QUILT session? Is there a correlation between question level and average gains in 
mathematics achievement test scores? 

As shown in Figure 3, for the classrooms observed in the present study, about 20% of the 
questions, on average, were found to be recall questions and about 74% were categorized as use 
questions. Only slightly more than 1% of the questions were higher cognitive level evaluation 
and synthesis questions, and about 3.5% of the questions were asked to check for understanding 
of procedures and routines. 
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Figure 3. Percentage of different types of questions asked by teachers 



Average gains in mathematics achievement scores were not found to be related to the 
percentage of recall questions posed, r = -.009, p = .962, the percentage of use questions, r = - 
.077, p = .695, or the percentage of evaluation/synthesis questions, r = -.016, p = .934. The 
correlation between mathematics achievement and the percentage of questions asked to check for 
understanding of procedures and routines was larger, r = .293, but not significant, p =.130. In 
sum, the analysis revealed no discernable relationship between the number of different types of 
questions teachers asked and the average gains on the Measures of Academic Progress in 
mathematics achievement. 

Research Question 5: Who responds most often to questions during a QUILT 
session? Is there a correlation between respondent and average gains in 
mathematics achievement test scores? 

The findings revealed that most questions teachers asked were answered by the students 
to whom they were directed. On average, this occurred 86.38% of the time. More than 12% of 
the time, the whole class responded to the teacher’s questions. Less than 1% of the time, the 
teacher responded to his/her own question, there was no response, or it could not be determined 
who responded. These outcomes are summarized in Figure 4. 
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Figure 4. Percentage of answers given by the teacher, the whole class, the designated 
student, an unknown student, or “no response” 



Average gains in mathematics achievement scores were not found to be related to the 
percentage of answers to teacher questions by designated students, r = -,149,p = .448. Thus, the 
desired teacher QUILT behavior was not found to be related to average mathematics gains on the 
Measures of Academic Progress. 

Research Question 6: What type of response is common during a QUILT session? Is 
there a correlation between the level of response and average gains in mathematics 
achievement test scores? 

On average, most of the responses given to the questions the teacher asked were use 
responses (73.49%). That is consistent with the fact that most questions were use questions 
(74.2%). Recall was the next most common type of response, on average (21.77%). This finding 
was consistent with the fact that 20.9% of the questions teachers posed were recall questions. 
There were very few of other types of responses given: evaluation and syntheses — 1.23%, 
unknown — 1.80%, and no response — 1.72%. There were no clarification responses recorded in 
any of the questioning sessions. These findings are summarized in Figure 5. 
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Figure 5. Percentage of answers of different types given in responses to teacher questions 



Average gains in mathematics achievement scores were not found to be related to the 
percentage of use questions, r = -.077, p = .695. This can be interpreted to mean that asking use 
questions and getting use answers is not related to the average gains on the Measures of 
Academic Progress in mathematics achievement that were observed. 

Research Question 7: Is Wait Time II used during a QUILT session? What is the 
correlation between Wait Time II and average gains in mathematics achievement 
test scores? 

Teachers gave no Wait Time II 95.81% of the time, as depicted in Figure 6. In almost all 
instances, as quickly as an answer to a question was given, the teacher immediately moved on to 
ask another question, to redirect the questions, and so forth. 



12 





3 sec 



Wait Time II (in seconds) 



Figure 6. Percentage of responses with no Wait Time II to more than 3 seconds 



The correlation between average Wait Time II and gain in mathematics achievement 
scores was not significant, r = -T93,p = .326, primarily because the teachers exhibited little, if 
any. Wait Time II. 

Research Question 8: How do teachers typically react to student answers during a 
QUILT session? Does teacher reaction appear to affect average gains in 
mathematics achievement test scores? 

The average number of questions teachers posed during the teaching sessions observed 
was approximately 42. As the students answered those questions, a variety of different types of 
teacher behaviors were observed in response to the students’ answers. These responses are 
summarized in Figure 7. Because teachers gave multiple types of responses to students’ answers, 
the total number of teacher responses is larger than the total number of questions asked and the 
total number students’ answers. 



13 




Teacher Response 



Figure 7. Average number of different types of teacher responses to students’ answers to 
teacher questions 



The average number of the different types of teacher responses to student answers for 
each teacher was examined in comparison to average gain in mathematics achievement for each 
teacher’s classroom. The QUILT model is based on the notion that variety of feedback is 
appropriate and that there tends to be an overuse of positive or surface answers such as “good 
answer” and “good try.” The QUILT model advocates increased use of corrective feedback, 
cues, and probes. Examination of the correlations between gain in student achievement and the 
average number of the various teacher behaviors revealed that only the average number of 
negative feedback responses was positively related to mathematics achievement, r = .382, p = 
.045, r^=.15. 

Research Question 9: How do students typically react to answers during a QUILT 
session? Does student reaction appear to affect average gains in mathematics 
achievement test scores? 

Students’ reactions to teachers’ questions, other than to answer them, were minimal. As 
shown in Figure 8, the most frequent student response was to add on to answers given by other 
students, but that response occurred only about 5% of the time. Other reactions were observed 
even less often. 
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Figure 8. Percentage of different types of student responses to teacher questions 



None of the average percentages of the students’ reactions were found to be significantly 
related to gains in mathematics achievement, p>.05. 
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Conclusions 



The findings of this pilot study indicate that teachers are regularly implementing some 
QUILT questioning behaviors as part of the instructional strategies being used in their 
mathematics instruction. However, this pilot study provided no direct evidence that these 
behaviors are systematically related to gains in mathematics achievement. The learnings gained 
from the pilot study will guide the development and implementation of an experimental efficacy 
study of the relationship between the use of the QUILT instructional strategy and student 
achievement. 



QUILT Behaviors 

The teachers in this pilot study were recorded implementing aspects of the instructional 
lessons observed. For instance, teachers exhibited a Wait Time I of more than 3 seconds about a 
third of the time — a behavior that would be expected to generate longer student responses, 
answers at higher evaluation and synthesis levels, and more student questions. Furthermore, 
more than 70% of the questions posed in the sessions observed were use questions. In contrast, 
(75% to 80% of teacher questions in a typical classroom are recall questions (Dillon, 1988; Gall, 
1984). Students in QUILT classrooms were designated as respondents after the teacher posed the 
question about 60% of the time (instead of before as typically done by most teachers). Teachers 
were also observed not repeating answers to questions very often. Not repeating answers 
facilitates students’ paying greater attention to and showing increased respect for their 
classmates’ responses. 

It was also observed that other QUILT behaviors were not frequently demonstrated 
during the instructional lessons. For example, the teachers exhibited almost no Wait Time II; 
they generally reacted immediately to a student response before providing feedback, allowing 
time for student questions, or making another instructional move. They were also rarely observed 
posing one question to more than one student (i.e., redirecting questions)-or probing or 
prompting to scaffold student responses. 

The pilot study demonstrated that QUILT coding system is usable to capture QUILT 
questioning behaviors of videotaped instructional lessons where QUILT is being used. While 
time consuming to accurately capture QUILT questioning behaviors, it is a highly reliable 
system. 



Gains in Mathematics Achievement 

No relationship was found between gains in mathematics achievement as assessed using 
the Measures of Academic Progress (Northwest Evaluation Association, 2004) and use of 
QUILT behaviors. This finding may be due in part to the fact that QUILT is a dynamic 
instructional technique composed of discrete, but interrelated, components that, when 
disaggregated, do not align with the goals of the Measure of Academic Progress Progress or the 
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levels of student cognition drawn on by questions posed by teachers as they implemented 
QUILT.. 



Next Steps 

The implementation of a well controlled, randomized control trial is needed to examine 
the efficacy of QUILT as an effective instructional technique. Based on the pilot study, the 
efficacy study should possess three key features. 

First, all teachers participating in the study should receive the same QUILT training at the 
same time provided by the same trainers. One of the sources of potential confounding in the pilot 
study was different teachers received training from different trainers at different times. 

Second, the fidelity of the implementation of QUILT questioning techniques should be 
systematically monitored to assure teachers do implement QUILT as designed and there is not 
“slippage” of implementation over time. The pilot study did not include a formal assessment of 
implementation fidelity. Therefore, it is not known whether QUILT behaviors “not often seen” 
by researchers in the pilot study (e.g.. Wait Time II) were just not commonly used by the 
teachers or if it was a differential training issue. 

Third, the instrument(s) used to assess student achievement should be one(s) that are sensitive to 
outcomes expected as the result of the use of QUILT (i.e., evidence of divergent thinking when 
problem solving) and that have evidence of being related to state assessments of student 
achievement. The Measures of Academic Progress mathematics test used in the pilot study by 
the cooperating district is an assessment of achievement in mathematics that is not known to be 
related to the state’s assessment of mathematics achievement of students and may or may not 
assess the use higher cognitive levels of thinking that are the goal associated with the 
implementation of QUILT. 
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Appendix 



Description of QUILT Program 



QUILT stands for Questioning and Understanding to Improve Learning and Thinking. As 
a yearlong professional development program for classroom teachers, QUILT applies research 
on learning, listening, and teacher effectiveness and is designed to enhance classroom 
questioning techniques and increase student learning and thinking. 

QUILT was developed to help educators recognize the relationship that exists between 
questioning practices and student learning outcomes, know the characteristics of effective 
classroom questioning, assess personal questioning practices and behaviors, use effective 
questioning and practices, and teach students how to question effectively (Walsh & Sattes, 

2004). 



Training in QUILT consists of four components: 

1. Induction training: Participants learn about effective questioning techniques during 
a four-day introductory training period, which includes a presentation of knowledge 
and theory, demonstration of behavior and skills, and application of the process. 

2. Collegiums: Once trained, participants meet periodically at their respective school 
sites to learn, share, and interact about the questioning behaviors targeted for 
practice and improvement. 

3. Partnering: Teams of educators, trained to use QUILT, participate in ongoing 
support activities at their respective schools, where teachers observe and are 
observed by their partners while engaged in the process. 

4. Classroom Application: During their involvement in the program, participants 
practice skills in the classroom and teach students about effective questioning 
practices. 

Schools, districts, or states usually send teams of educators to a national QUILT 
professional development workshop for training in the four components of OUILT listed above. 
Teams trained in QUILT typically return to their schools, districts, or states and provide an 
abbreviated version of the training to colleagues at their respective educational institutions where 
the above components are actualized and QUILT behaviors are introduced as an ongoing 
teaching/learning strategy (Walsh & Sattes, 1995). 



Walsh, J., & Sattes, B. (1995). QUILT. Charleston, WV: ALL. 

Walsh, J., & Sattes, B. (2004). QUILT teacher manual. Charleston, WV: ALL. 
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